Multi-layer qos management in a distributed computing environment

ABSTRACT

A system for multi-layer quality of service (QoS) management in a distributed computing environment includes a management node hosting a workload scheduler operable to receive a workload and identify a workload QoS class for the workload. The system also includes a plurality of distributed compute nodes. A workload scheduler is operable to schedule running of the workload on the compute nodes. The workload scheduler is operable to translate the workload QoS class to a storage level QoS class and communicate the storage level QoS class to a workload execution manager of the compute nodes. The workload scheduler communicates the storage level QoS class to one or more storage managers where the storage managers manage storage resources, and the storage managers are operable to extend the storage level QoS class to the storage resources to support the workload QoS class.

BACKGROUND

In cluster computing or in a distributed computing environment, acompute cluster is a set of computers connected over a network withresource usage within the cluster coordinated by a workload or resourcemanager. Typically, a user submits a job (a request to run an instanceof an application) to the resource manager. The resources required torun the job may be specified by the user with the job submission orallocated as needed by the resource manager. The resource managerassigns idle resources to the job when available, and runs the job.

The workload of each compute cluster may be managed by a workloadscheduler. In some cases, a resource may be shared among multipleindependent clusters. Thus, certain resources may need to be allocatedacross the different compute clusters of the organization. Indistributed application scenarios, the application may be data-intensiveand compute-intensive. For example, applications are often hosted in amulti-tenancy environment where distributed computers, network, andstorages are shared by other applications so as to minimizeinfrastructure and management costs. Some of these application may alsohave constraints such as response times, such as interactive or nearreal-time decision making applications used in stock purchases andpersonalized recommendations for mobile device users.

BRIEF SUMMARY

According to one aspect of the present disclosure a system and techniquefor multi-layer quality of service (QoS) management in a distributedcomputing environment is disclosed. The system includes a managementnode hosting a workload scheduler operable to receive a workload andidentify a workload QoS class for the workload. The system also includesa plurality of distributed compute nodes. A workload scheduler isoperable to schedule running of the workload on the compute nodes. Theworkload scheduler is operable to translate the workload QoS class to astorage level QoS class and communicate the storage level QoS class to aworkload execution manager of the compute nodes. The workload schedulercommunicates the storage level QoS class to one or more storage managerswhere the storage managers manage storage resources, and the storagemanagers are operable to extend the storage level QoS class to thestorage resources to support the workload QoS class.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

For a more complete understanding of the present application, theobjects and advantages thereof, reference is now made to the followingdescriptions taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is an embodiment of a network of data processing systems in whichthe illustrative embodiments of the present disclosure may beimplemented;

FIG. 2 is an embodiment of a data processing system in which theillustrative embodiments of the present disclosure may be implemented;

FIG. 3 is a diagram illustrating an embodiment of a distributedcomputing environment in which illustrative embodiments of a system formulti-layer quality of service (QoS) management in a distributedcomputing environment according to the present disclosure may beimplemented; and

FIG. 4 is a flow diagram illustrating an embodiment of a method formulti-layer quality of service (QoS) management in a distributedcomputing environment according to the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure provide a method, system andcomputer program product for multi-layer QoS management in a distributedcomputing environment. Embodiments of the present disclosure areconfigured to dynamically manage and classify workload and storage QoSacross compute, network and storage layers for data- andcompute-intensive applications running in distributed computingenvironment. For example, QoS settings are identified and mapped tovarious resource managers of the various layers of the environment forextending such QoS settings to respectively controlled layer resources.Embodiments of the present disclosure also proactively adjust QoSsettings in various layers of the environment for various QoS classesaccording to workload scheduling policies and demands. Embodiments ofthe present disclosure may also schedule workloads and accommodate dataread/write requests to avoid and/or alleviate hot spots of a storage QoSclass. Thus, for example, in some embodiments, the method and techniqueincludes: receiving a workload to run in a distributed computingenvironment; identifying a workload quality of service (QoS) class forthe workload; translating the workload QoS class to a storage level QoSclass; scheduling the workload to run on a compute node of theenvironment; communicating the storage level QoS class to a workloadexecution manager of the compute node; communicating the storage levelQoS class to one or more storage managers of the environment, thestorage managers managing storage resources in the environment; andextending, by the storage managers, the storage level QoS class to thestorage resources to support the workload QoS class.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

With reference now to the Figures and in particular with reference toFIGS. 1-2, exemplary diagrams of data processing environments areprovided in which illustrative embodiments of the present disclosure maybe implemented. It should be appreciated that FIGS. 1-2 are onlyexemplary and are not intended to assert or imply any limitation withregard to the environments in which different embodiments may beimplemented. Many modifications to the depicted environments may bemade.

FIG. 1 is a pictorial representation of a network of data processingsystems in which illustrative embodiments of the present disclosure maybe implemented. Network data processing system 100 is a network ofcomputers in which the illustrative embodiments of the presentdisclosure may be implemented. Network data processing system 100contains network 130, which is the medium used to provide communicationslinks between various devices and computers connected together withinnetwork data processing system 100. Network 130 may include connections,such as wire, wireless communication links, or fiber optic cables.

In some embodiments, server 140 and server 150 connect to network 130along with data store 160. Server 140 and server 150 may be, forexample, IBM® Power Systems™ servers. In addition, clients 110 and 120connect to network 130. Clients 110 and 120 may be, for example,personal computers or network computers. In the depicted example, server140 provides data and/or services such as, but not limited to, datafiles, operating system images, and applications to clients 110 and 120.Network data processing system 100 may include additional servers,clients, and other devices.

In the depicted example, network data processing system 100 is theInternet with network 130 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. At the heart of the Internet is a backbone ofhigh-speed data communication lines between major nodes or hostcomputers, consisting of thousands of commercial, governmental,educational and other computer systems that route data and messages. Ofcourse, network data processing system 100 also may be implemented as anumber of different types of networks, such as for example, an intranet,a local area network (LAN), or a wide area network (WAN). FIG. 1 isintended as an example, and not as an architectural limitation for thedifferent illustrative embodiments.

FIG. 2 is an embodiment of a data processing system 200 such as, but notlimited to, client 110 and/or server 140 in which an embodiment of asystem for multi-layer QoS management in a distributed computingenvironment according to the present disclosure may be implemented. Inthis embodiment, data processing system 200 includes a bus orcommunications fabric 202, which provides communications betweenprocessor unit 204, memory 206, persistent storage 208, communicationsunit 210, input/output (I/O) unit 212, and display 214.

Processor unit 204 serves to execute instructions for software that maybe loaded into memory 206. Processor unit 204 may be a set of one ormore processors or may be a multi-processor core, depending on theparticular implementation. Further, processor unit 204 may beimplemented using one or more heterogeneous processor systems in which amain processor is present with secondary processors on a single chip. Asanother illustrative example, processor unit 204 may be a symmetricmulti-processor system containing multiple processors of the same type.

In some embodiments, memory 206 may be a random access memory or anyother suitable volatile or non-volatile storage device. Persistentstorage 208 may take various forms depending on the particularimplementation. For example, persistent storage 208 may contain one ormore components or devices. Persistent storage 208 may be a hard drive,a flash memory, a rewritable optical disk, a rewritable magnetic tape,or some combination of the above. The media used by persistent storage208 also may be removable such as, but not limited to, a removable harddrive.

Communications unit 210 provides for communications with other dataprocessing systems or devices. In these examples, communications unit210 is a network interface card. Modems, cable modem and Ethernet cardsare just a few of the currently available types of network interfaceadapters. Communications unit 210 may provide communications through theuse of either or both physical and wireless communications links.

Input/output unit 212 enables input and output of data with otherdevices that may be connected to data processing system 200. In someembodiments, input/output unit 212 may provide a connection for userinput through a keyboard and mouse. Further, input/output unit 212 maysend output to a printer. Display 214 provides a mechanism to displayinformation to a user.

Instructions for the operating system and applications or programs arelocated on persistent storage 208. These instructions may be loaded intomemory 206 for execution by processor unit 204. The processes of thedifferent embodiments may be performed by processor unit 204 usingcomputer implemented instructions, which may be located in a memory,such as memory 206. These instructions are referred to as program code,computer usable program code, or computer readable program code that maybe read and executed by a processor in processor unit 204. The programcode in the different embodiments may be embodied on different physicalor tangible computer readable media, such as memory 206 or persistentstorage 208.

Program code 216 is located in a functional form on computer readablemedia 218 that is selectively removable and may be loaded onto ortransferred to data processing system 200 for execution by processorunit 204. Program code 216 and computer readable media 218 form computerprogram product 220 in these examples. In one example, computer readablemedia 218 may be in a tangible form, such as, for example, an optical ormagnetic disc that is inserted or placed into a drive or other devicethat is part of persistent storage 208 for transfer onto a storagedevice, such as a hard drive that is part of persistent storage 208. Ina tangible form, computer readable media 218 also may take the form of apersistent storage, such as a hard drive, a thumb drive, or a flashmemory that is connected to data processing system 200. The tangibleform of computer readable media 218 is also referred to as computerrecordable storage media. In some instances, computer readable media 218may not be removable.

Alternatively, program code 216 may be transferred to data processingsystem 200 from computer readable media 218 through a communicationslink to communications unit 210 and/or through a connection toinput/output unit 212. The communications link and/or the connection maybe physical or wireless in the illustrative examples.

The different components illustrated for data processing system 200 arenot meant to provide architectural limitations to the manner in whichdifferent embodiments may be implemented. The different illustrativeembodiments may be implemented in a data processing system includingcomponents in addition to or in place of those illustrated for dataprocessing system 200. Other components shown in FIG. 2 can be variedfrom the illustrative examples shown. For example, a storage device indata processing system 200 is any hardware apparatus that may storedata. Memory 206, persistent storage 208, and computer readable media218 are examples of storage devices in a tangible form.

FIG. 3 is an illustrative embodiment of a system 300 for multi-layer QoSmanagement in a distributed computing environment (e.g., a clusteredcomputing environment). Various components or nodes of system 300 may beimplemented on data processing systems or platforms such as, but notlimited to, servers 140 and/or 150, clients 110 and/or 120, or at otherdata processing system locations. In FIG. 3, system 300 illustrates anexemplary computing architecture 302 where a plurality of dataprocessing nodes are configured to communicate with one another and/orshare resources across the computing environment. Each node may includeone or more linked machines or “hosts” which are configured to provideresources such as CPU time, memory, database storage, software licenses,and computing capabilities. A host may be any machine capable ofproviding resources, such as a personal computer (PC), a server, orother type of computing device.

In the embodiment illustrated in FIG. 3, system 300 includes amanagement node 310 and compute/data nodes 312 (e.g., compute/data nodes312 ₁-312 _(n)). Management node 310 includes a workload scheduler 314that is configured to receive workloads from one or more clients/users316. In the illustrated embodiment, a single management node 310 with asingle workload scheduler 314 is depicted; however, it should beunderstood that multiple management nodes 310 may be employed each withone or more workload schedulers 314. Workload scheduler 314 may evaluatesubmitted workloads and perform various resource scheduling andallocation decisions for executing/processing the workloads. Forexample, workload scheduler may manage the resources in the cluster,including compute resources (e.g. CPU and memory), storage resources,and network resources, and schedule the workloads to run on compute/datanodes 312. Each compute/data node 312 may also comprise a workloadexecution manager 318 (e.g., workload execution managers 318 ₁-318 _(n))that performs various resource scheduling and allocation decisions localto a respective compute/data node 312 for running or executingworkloads. Workload scheduler 314 and/or workload execution manager 318may be implemented in any suitable manner using known techniques thatmay be hardware-based, software-based, or some combination of both. Forexample, workload scheduler 314 and/or workload execution manager 318may comprise software, logic and/or executable code for performingvarious functions as described herein (e.g., residing as software and/oran algorithm running on a processor unit, hardware logic residing in aprocessor or other type of logic chip, centralized in a singleintegrated circuit or distributed among different chips in a dataprocessing system).

In FIG. 3, system 300 also includes a distributed storage system 320including local data stores 322 (e.g., local data stores 322 ₁-322 _(n))associated with respective compute/data nodes 312, and data server nodes324 (e.g., data server nodes 324 ₁-324 _(n)) each having available datastores 326 (e.g., data stores 326 ₁-326 _(n)) that may be optionallydedicated. A storage management node 330 includes a distributed storagemanager 332 that manages shared storage via the local storage (e.g.,local data stores 322) through a local storage manager 340 on respectivecompute/data node 312 (e.g., local storage managers 340 ₁-340 _(n)) andvia data stores 326 through respective data server nodes 324. Storagemanager 332, by coordinating with local storage managers 340, provides aglobal picture of a single storage system to serve data read/writerequests on the local and shared storages for jobs and tasks. Storagemanager 322 may be implemented in any suitable manner using knowntechniques that may be hardware-based, software-based, or somecombination of both. For example, storage manager 322 may comprisesoftware, logic and/or executable code for performing various functionsas described herein (e.g., residing as software and/or an algorithmrunning on a processor unit, hardware logic residing in a processor orother type of logic chip, centralized in a single integrated circuit ordistributed among different chips in a data processing system).

One or more networks 130 (e.g., networks 130 ₁ and 130 ₂) connect thenodes in the cluster. For data connections, a network (e.g., network 130₁) may connect each compute/data node 312 so that nodes 312 can exchangedata among each other. There may also be a network (e.g., network 130 ₂)connecting compute/data nodes 312 to the dedicated data server nodes 324to read/write data in the shared data stores 326. Another network (e.g.a storage area network (SAN)) among the dedicated data server nodes 326to accommodate high performance data replication and striping.

Embodiments of the present disclosure enable the dynamical managementand classification of workload and storage QoS across compute, networkand storage layers for applications running in a distributed computingenvironment. Embodiments of the present disclosure further provideconsistent and collaborative QoS classification and mapping so that theclassification and isolation are performed in a coordinated andeffective fashion, despite the heterogeneous specifics amongst systemcomponents. For example, the sharing of resources in a distributedcomputing environment may create complex resource interference andcontention scenarios, thereby making it difficult to provide desired QoSfor high priority applications requiring interactive or near real-timedecision processes. For example, workloads of interactive applicationsrequire fast responses to users in the range of seconds.

The QoS of a resource (e.g., network or storage) is usually representedas a class, policy tag, or priority number provided by a resourcemanager (e.g. a file system) to the application workload layer forcontrolling what performance criteria the resource layer should deliverto a workload in terms of resource scheduling priority, input/outputoperations per second (TOPS), bandwidth, latency guarantee, limit, etc.According to the present disclosure, on top of the resource layer, suchclassification can also be managed in the workload management layer sothat a high-level QoS in terms of priority, bandwidth/throughputisolation or limits, and latency requirements for different workloadscan be consistently specified, translated and propagated from theworkload scheduler 314 into resource managers/controllers in adistributed computing environment. Accordingly, the resource layerthereby provides the required QoS support for these workloads, oftenusing exiting actuators available at different resources such as CPU,memory, cache, storage input/output (I/O) and/or network I/O.

A workload can be either a service workload, or a job, or a task in ajob. A service workload submitted from a user/client 316 may comprise along running or always running software program such as a web server ora database server. Compared to a service workload, a job or a task hasrelatively short life cycle. A job submitted from a user/client 316 caninclude multiple groups of tasks. Tasks in a group have the sameresource and QoS requirements that can be specified either in anapplication configuration profile or in the job submission command line.Examples of QoS requirements can be high/medium/low data throughputs orbandwidths, high/medium/low latency requirements, and/or differentpriorities or business importance. Tasks among groups may also have dataand/or execution dependencies or dependencies among work steps oractivities in a workflow job. Different jobs or different groups oftasks in a job may have different QoS requirements with given QoSclasses (e.g., one may require high data throughput/bandwidth, but isfine with high latency while another may require low latency, but isfine with low data throughput/bandwidth).

In operation in response to obtaining a QoS requirement or class of aworkload (e.g., from either a command line or configuration), workloadscheduler 314 translates the QoS requirement into a corresponding QoSclass for each involved resource layer, such as storage and/or network,according to the configurations in the systems. Also, since distributedstorage system 320 controls the internal network 130 within the storagesystem 320, the distributed storage manager 332 and local storagemanager 340 set necessary network QoS classes in the internal network130 according to the storage QoS class requested by workload scheduler314. For example, storage QoS classes can be set through an ionice,cgroups or other type of command (e.g., depending on the operatingsystem). Network QoS classes can be set through traffic controller (tc)commands as well as various network software commands, protocols orconfigurations provided from different network vendors.

Embodiments of the present disclosure also proactively adjust QoSsettings (e.g., reserved resources such as buffer sizes, bandwidths,priorities, tokens, etc.) for various QoS classes/zones/bands accordingto workload demands. For example, to guarantee a QoS for variousapplication workloads, some resources such as memory buffers, bandwidthsand token bucket sizes are reserved for various QoS classes/zone/bands(e.g., high/medium/low throughputs or latencies). Since resources innetwork and storage layers are valuable, reserving such resources allthe time on every data node and network switch may not result inefficient utilization of such resources. If idle resources of a certainclass may be freely used by workloads of any other classes in any order,these idle resources are not well managed regarding to the high levelworkload scheduling policies. Embodiments of the present disclosureenable end-to-end QoS management between workload scheduling andresource layers. For example, workload scheduler 314 is aware of whatworkloads are running and pending in a cluster, what data nodes or agroup of nodes are used or will be used by workloads of an applicationbased on the workload scheduling policies and execution calendar or timewindows. According to scheduling information and policies, workloadscheduler 314 instructs the resource managers in the storage and networklayers to dynamically set, change, or free up these resources and adaptthe settings of various QoS classes/zones/bands in a workload-drivenfashion. For example, when there is no current and pending workload fora QoS class/zone/band, the workload scheduler 314 can instruct theresource manager (e.g., distributed storage manager 332) to free upreserved resources from this QoS class/zone/band so that the resourcescan be well managed and allocated to other QoS classes/zones/bands thathave current and pending workloads according to the high level workloadscheduling policies.

Embodiments of the present disclosure also intelligently scheduleworkloads and accommodate data read/write requests to avoid or alleviatehot spots of a storage QoS class. For example, in distributedenvironments, a data block or file may be replicated on multiple datanodes for parallel access performance and fault tolerance. A data accesshot spot may be caused on a data node if too many workloads need to readand/or write data on this node in the same QoS class, or if the overallperformance of a data node is bogged down for various reasons. A hotspot is related to a node, or even a network switch node. The hot spotmay also be related to a specific QoS class or all the QoS classes onthe node. Placing new loads on a hot spot node may slow down not onlythe new loads but also the existing loads.

In response to detecting a hot spot node by storage managers 332 or 340,the hot spot can be used by the storage layer to accommodate data readrequests with other data nodes that have the replicas of the requesteddata. For example, if a data block is being written, the storage layer(e.g., storage managers 332 or 340) can avoid this hot spot node whenchoosing which nodes to write and replicate the data block. The hot spotinformation can also be used by workload scheduler 314 to avoid a hotspot node, but use other nodes with the same replicas when doingdata-locality-aware scheduling, or even defer scheduling some workloadsthat have to use the hot spot node. The storage layer (e.g., storagemanagers 332 or 340) and workload scheduler 314 will make thesedecisions according to whether a hot spot is just specific to a QoSclass or all the QoS classes on a node. After a hot spot node has beencooled down, the node can be used as a normal node again by the storagelayer and workload scheduler 314. Further, if a data block is frequentlyread and highly demanded by the current and pending workloads whichcause or may potentially cause hot spots in a QoS class, workloadscheduler 314 and storage managers 332 and/or 340 work together toautomatically increase replicas of this data block. Alternatively, hotspots can also be alleviated for selected QoS classes by sacrificing theperformance of some other QoS classes sharing the same hot spot. The QoSclasses selected for protection may have higher priority/importance orbe more latency sensitive than the QoS classes chosen for sacrifice. Inone embodiment, I/O requests from the sacrificed QoS classes may berate-limited so as to ease the load on the hot spot.

In operation, QoS classes/priorities are defined at theapplication/workload levels (e.g., batch/throughput (reliable, largedata volume), interactive (<10s latency, pause and run spikes, smalldata exchanges), real-time (as soon as possible, streaming), etc.).Workload scheduler 314 maps the high-level application/workload QoSclasses/priorities to the low-level storage and/or network level QoSclasses/priorities which are managed and controlled by local storagemanagers 340 and/or distributed storage manager 332. These low-level QoSclasses/priorities can be set via command line or applicationprogramming interfaces of storage managers or operating systems such asionice, cgroups, traffic controller (tc), etc. The low-level QoSclasses/priorities are extended by distributed storage manager 332 todata server nodes 324 and by local storage managers 340 forcorresponding data stores 322 to support distributed and local blockstorages so that distributed storage manager 332 and local storagemanager 340 can extract the settings of low-level QoS classes/prioritiesfor distributed block storage from the operating system. Based on thesettings for the distributed block storages, distributed storage manager332 and/or local storage managers 340 can manage and control QoS of thedistributed storage systems 320 as well as the storage network insidethe system 320.

In response to workload scheduler 314 scheduling a workload to run inthe cluster (e.g., by one or more compute/data nodes 312), workloadscheduler 314 translates the workload's QoS classes/priorities to thestorage/network level QoS classes/priorities, and then passes thestorage/network level QoS classes/priorities to workload executionmanagers 318 on respective compute/data nodes 312, as well as todistributed storage manager 332 if needed. Workload execution managers318 further pass the storage/network level QoS classes/priorities torespective local storage managers 340.

Workload scheduler 314 also checks the QoS requirements of currentlypending and running workloads and communicates to distributed storagemanager 332 the number of running workloads in the cluster and thenumber of pending workloads that are in workload queues for specifiedQoS classes. Distributed storage manager 332 and/or local storagemanagers 340 adjust/re-adjust reserved resource distributions amongdifferent QoS classes on demand (e.g., in response to instructionsreceived from workload scheduler 314 based on workload schedulingpolicies and demands).

Distributed storage manager 332 and/or local storage managers 340 assigndisk I/O bandwidth allocation and/or cache sizes to storage I/O streamsfrom different QoS classes. According to the QoS classes of tags, forexample, storage I/O streams with higher priority QoS classes are givenmore disk bandwidth allocation and/or cache sizes. The storage I/Ostreams of low priority QoS classes can be rate-limited on their I/Obandwidth and cache sizes.

Distributed storage manager 332 and/or local storage managers 340 alsomonitor for hot spots (i.e., heavily loaded nodes, heavily loaded QoSclasses, frequently accessed data blocks) in storage/network layers. Inresponse to detecting a hot spot by distributed storage manager 332and/or local storage managers 340, distributed storage manager 332and/or local storage managers 340 report the hot spots to workloadscheduler 314. Workload scheduler 314 then schedules workloads to avoidhot spots. Workload scheduler 314 may inform distributed storage manager332 and/or local storage managers 340 which workloads/jobs/tasks havebeen rescheduled to other nodes. Distributed storage manager 332 and/orlocal storage managers 340 may also replicate data blocks to non-hotspots nodes where the newly re-created workloads/jobs/tasks will be run.Distributed storage manager 332 and/or local storage managers 340 mayalso move some resources from less loaded nodes and QoS classes toheavily loaded nodes and QoS classes.

FIG. 4 is a flow diagram illustrating an embodiment of a method formulti-layer QoS management in a distributed computing environment. Themethod begins at block 402, where workload scheduler 314 received aworkload from a user/client 316. At block 404, workload scheduler 314identifies workload QoS classes/priorities for the workload (e.g.,batch/throughput (reliable, large data volume), interactive (<10slatency, pause and run spikes, small data exchanges), real-time (as soonas possible, streaming), etc.). At block 406, workload scheduler 314translates and/or maps the workload QoS classes/priorities to thestorage and/or network level QoS classes/priorities. At block 408,workload scheduler 314 schedules and distributes the workload torespective compute/data nodes 312. At block 410, workload scheduler 314communicates/transmits the storage/network layers QoS classes/prioritiesto workload execution managers 318 of respective compute/data nodes 312as well as to distributed storage manager 332, and workload executionmanagers 318 communicate/transmit the storage/network layers QoSclasses/priorities to respective local storage managers 340.

At block 412, distributed storage manager 332 and/or local storagemanagers 340 extend the storage/network layers QoS classes/priorities torespective distributed storage/network layer resources (e.g., datastores 322, network 130 ₂, data server nodes 324, etc.). At block 414,workload scheduler 314 checks the workload scheduling policies and QoSrequirements of currently pending and running workloads. At block 416,workload scheduler 314 requests distributed storage manager 332 and/orlocal storage managers 340 to adjust reserved storage resourcesaccording to the workload scheduling policies and the number of runningworkloads in the cluster and the number of pending workloads that are inworkload queues for various QoS classes. At block 418, distributedstorage manager 332 and/or local storage managers 340 adjust/re-adjustreserved resource distributions among different QoS classes based onrequests from the workload scheduler according to workload schedulingpolicies and demands. For example, distributed storage manager 332and/or local storage managers 340 may redistribute/reallocate disk I/Obandwidth and/or cache sizes to storage I/O streams from different QoSclasses.

At block 420, distributed storage manager 332 and/or local storagemanagers 340 monitor for hot spots (i.e., heavily loaded nodes, heavilyloaded QoS classes, frequently accessed data blocks) in storage/networklayers. At block 422, distributed storage manager 332 and/or localstorage managers 340 notify workload scheduler 314 of detected hotspots. At block 424, workload scheduler 314 schedules/re-schedulesworkloads to avoid hot spots. At block 426, workload scheduler 314informs/notifies distributed storage manager 332 and/or local storagemanagers 340 which workloads/jobs/tasks have been rescheduled to othernodes. At block 428, distributed storage manager 332 and/or localstorage managers 340 may move some resources from less loaded nodes andQoS classes to heavily loaded nodes and QoS classes. Distributed storagemanager 332 and/or local storage managers 340 may also replicate datablocks to non-hot spots nodes where the newly re-createdworkloads/jobs/tasks will be run.

Thus, as indicated above, interactive and batch workloads ofdata-intensive applications store a majority of data in distributedstorage systems, retrieve the data from memory on compute nodescross-network, analyze the data in CPU, and then either send the resultsback to interactive users or write the results back to the distributedstorage systems cross-network. Therefore, embodiments of the presentdisclosure are configured to dynamically manage and classify workloadand storage QoS across compute, network and storage layers for data- andcompute-intensive applications running in distributed computingenvironment with consistent and collaborative QoS classification andmapping across the different layers of the environment so that theclassification and isolation are done in a coordinated and effectivefashion. Embodiments of the present disclosure also proactively adjustQoS settings of reserved resources in various layers of the environmentfor various QoS classes/zones/bands according to workload schedulingpolicies and demands, and schedule workloads and accommodate dataread/write requests to avoid and/or alleviate hot spots of a storage QoSclass.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present disclosure has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the disclosure in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the disclosure. Theembodiment was chosen and described in order to best explain theprinciples of the disclosure and the practical application, and toenable others of ordinary skill in the art to understand the disclosurefor various embodiments with various modifications as are suited to theparticular use contemplated.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

What is claimed is:
 1. A system, comprising: a management node having aprocessor coupled to a memory, the management node hosting a workloadscheduler operable to receive a workload and identify a workload qualityof service (QoS) class for the workload; and a plurality of distributedcompute nodes, the workload scheduler operable to schedule running ofthe workload on the compute nodes; and wherein the workload scheduler isoperable to: translate the workload QoS class to a storage level QoSclass; communicate the storage level QoS class to a workload executionmanager of the compute nodes; and communicate the storage level QoSclass to one or more storage managers, the storage managers managingstorage resources; and wherein the storage managers are operable toextend the storage level QoS class to the storage resources to supportthe workload QoS class.
 2. The system of claim 1, wherein: the workloadexecution manager is operable to communicate the storage level QoS classto a local storage manager, the local storage manager managing a storageresource local to the compute node; and wherein the workload scheduleris operable to communicate the storage level QoS class to a distributedstorage manager, the distributed storage manager managing non-local,distributed storage resources relative to the compute node.
 3. Thesystem of claim 1, wherein the workload scheduler is operable to: checkQoS requirements for running and pending workloads; and notify thestorage managers of a quantity of running and pending workloadsassociated with a particular QoS class.
 4. The system of claim 3,wherein the storage managers are operable to adjust reserved storageresource allocations in storage managers based on workload schedulingpolicies and resource demands as indicated by the workload scheduleraccording to workload scheduling policies and the quantity of runningand pending workloads associated with the particular QoS class.
 5. Thesystem of claim 1, wherein the storage managers are operable to monitorfor hot spots, wherein the hot spots indicate increased loading withparticular QoS classes.
 6. The system of claim 5, wherein the storagemanagers are operable to notify the workload scheduler of the hot spots,and wherein the workload scheduler is operable to schedule workloads toavoid the hot spots.
 7. The system of claim 5, wherein the storagemanagers are operable to notify the workload scheduler of the hot spots,and wherein the workload scheduler is operable to defer scheduling ofworkloads that will use the hot spots.
 8. The system of claim 5, whereinthe storage managers are operable to replicate data to non-hot spots andcause workloads to use data replicated to the non-hot spots instead ofthe data of the hot-spots.
 9. A computer program product for multi-layerquality of service (QoS) management in a distributed computingenvironment, the computer program product comprising a computer readablestorage medium having program instructions embodied therewith, theprogram instructions executable by a processor to cause the processor toperform a method comprising: receiving a workload to run in theenvironment; identifying a workload quality of service (QoS) class forthe workload; translating the workload QoS class to a storage level QoSclass; scheduling the workload to run on a compute node of theenvironment; communicating the storage level QoS class to a workloadexecution manager of the compute node; communicating the storage levelQoS class to one or more storage managers of the environment, thestorage managers managing storage resources in the environment; andextending, by the storage managers, the storage level QoS class to thestorage resources to support the workload QoS class.
 10. The computerprogram product of claim 9, wherein the program instructions areexecutable by the processor to perform the method further comprising:communicating the storage level QoS class by the workload executionmanager to a local storage manager, the local storage manager managing astorage resource local to the compute node; and communicating thestorage level QoS class to a distributed storage manager, thedistributed storage manager managing non-local, distributed storageresources relative to the compute node.
 11. The computer program productof claim 9, wherein the program instructions are executable by theprocessor to cause the processor to perform the method furthercomprising: checking workload scheduling policies and QoS requirementsfor running and pending workloads; and requesting the storage managersto adjust reserved storage resources according to workload schedulingpolicies and a quantity of running and pending workloads associated witha particular QoS class.
 12. The computer program product of claim 11,wherein the program instructions are executable by the processor toperform the method further comprising adjusting reserved storageresource allocation by the storage managers based on requests accordingto workload scheduling policies and the quantity of running and pendingworkloads associated with the particular QoS class.
 13. The computerprogram product of claim 9, wherein the program instructions areexecutable by the processor to cause the processor to perform the methodfurther comprising monitoring, by the storage managers, for hot spots,wherein the hot spots indicate increased loading with particular QoSclasses.
 14. The computer program product of claim 13, wherein theprogram instructions are executable by the processor to cause theprocessor to perform the method further comprising: notifying, by thestorage managers, a workload scheduler of the hot spots; and scheduling,by the workload scheduler, workloads to avoid the hot spots.
 15. Thecomputer program product of claim 13, wherein the program instructionsare executable by the processor to cause the processor to perform themethod further comprising: notifying, by the storage managers, aworkload scheduler of the hot spots; and deferring, by the workloadscheduler, scheduling of workloads that will use the hot spots.
 16. Thecomputer program product of claim 13, wherein the program instructionsare executable by the processor to cause the processor to perform themethod further comprising: replicating, by the storage managers, data tonon-hot spots; and causing, by the storage managers, workloads to usedata replicated to the non-hot spots instead of the data of thehot-spots.